Undertaking rapid evaluations during the COVID-19 pandemic: Lessons from evaluating COVID-19 remote home monitoring services in England

Introduction Rapid evaluations can offer evidence on innovations in health and social care that can be used to inform fast-moving policy and practise, and support their scale-up according to previous research. However, there are few comprehensive accounts of how to plan and conduct large-scale rapid evaluations, ensure scientific rigour, and achieve stakeholder engagement within compressed timeframes. Methods Using a case study of a national mixed-methods rapid evaluation of COVID-19 remote home monitoring services in England, conducted during the COVID-19 pandemic, this manuscript examines the process of conducting a large-scale rapid evaluation from design to dissemination and impact, and reflects on the key lessons for conducting future large-scale rapid evaluations. In this manuscript, we describe each stage of the rapid evaluation: convening the team (study team and external collaborators), design and planning (scoping, designing protocols, study set up), data collection and analysis, and dissemination. Results We reflect on why certain decisions were made and highlight facilitators and challenges. The manuscript concludes with 12 key lessons for conducting large-scale mixed-methods rapid evaluations of healthcare services. We propose that rapid study teams need to: (1) find ways of quickly building trust with external stakeholders, including evidence-users;(2) consider the needs of the rapid evaluation and resources needed;(3) use scoping to ensure the study is highly focused;(4) carefully consider what cannot be completed within a designated timeframe;(5) use structured processes to ensure consistency and rigour;(6) be flexible and responsive to changing needs and circumstances;(7) consider the risks associated with new data collection approaches of quantitative data (and their usability);(8) consider whether it is possible to use aggregated quantitative data, and what that would mean when presenting results, (9) consider using structured processes & layered analysis approaches to rapidly synthesise qualitative findings, (10) consider the balance between speed and the size and skills of the team, (11) ensure all team members know roles and responsibilities and can communicate quickly and clearly;and (12) consider how best to share findings, in discussion with evidence-users, for rapid understanding and use. Conclusion These 12 lessons can be used to inform the development and conduct of future rapid evaluations in a range of contexts and settings.

Undertaking rapid evaluations during the COVID-19 pandemic: Lessons from evaluating COVID-19 remote home monitoring services in England.

Introduction: Rapid evaluations can offer evidence on innovations in health and social care that can be used to inform fast-moving policy and practise, and support their scale-up according to previous research. However, there are few comprehensive accounts of how to plan and conduct large-scale rapid evaluations, ensure scientific rigour, and achieve stakeholder engagement within compressed timeframes. Methods: Using a case study of a national mixed-methods rapid evaluation of COVID-19 remote home monitoring services in England, conducted during the COVID-19 pandemic, this manuscript examines the process of conducting a large-scale rapid evaluation from design to dissemination and impact, and reflects on the key lessons for conducting future large-scale rapid evaluations. In this manuscript, we describe each stage of the rapid evaluation: convening the team (study team and external collaborators), design and planning (scoping, designing protocols, study set up), data collection and analysis, and dissemination. Results: We reflect on why certain decisions were made and highlight facilitators and challenges. The manuscript concludes with 12 key lessons for conducting large-scale mixed-methods rapid evaluations of healthcare services. We propose that rapid study teams need to: (1) find ways of quickly building trust with external stakeholders, including evidence-users; (2) consider the needs of the rapid evaluation and resources needed; (3) use scoping to ensure the study is highly focused; (4) carefully consider what cannot be completed within a designated timeframe; (5) use structured processes to ensure consistency and rigour; (6) be flexible and responsive to changing needs and circumstances; (7) consider the risks associated with new data collection approaches of quantitative data (and their usability); (8) consider whether it is possible to use aggregated quantitative data, and what that would mean when presenting results, (9) consider using structured processes & layered analysis approaches to rapidly synthesise qualitative findings, (10) consider the balance between speed and the size and skills of the team, (11) ensure all team members know roles and responsibilities and can communicate quickly and clearly; and (12) consider how best to share findings, in discussion with evidence-users, for rapid understanding and use. Conclusion: These 12 lessons can be used to inform the development and conduct of future rapid evaluations in a range of contexts and settings.

Early evidence of the development of primary care networks in England: a rapid evaluation study

BackgroundPrimary care networks are groups of general practices brought together in July 2019 to hold shared budgets and develop new services in response to NHS England (now NHS England and Improvement) policy. This study examined the early implementation of primary care networks, especially what has helped or hindered progress, how they operate in relation to pre-existing collaborations and issues for rural networks.ObjectiveTo produce early evidence about the development of primary care networks to inform policy for their future development. Our research questions were to investigate (1) the contextual background of primary care networks;(2) the rationale for general practices to enter into collaborations;(3) the early learning from establishing primary care networks;(4) barriers to and facilitators of effective collaboration across general practices;and (5) the likely future progress of primary care networks in the English NHS, including in the light of COVID-19.DesignA qualitative cross-comparative case study evaluation comprised four work packages: (1) a rapid evidence assessment;(2) a workshop with academics, policy experts and patient/public representatives;(3) interviews with stakeholders, observations of meetings, a survey and documentary analysis across four case study sites;and (4) analysis and synthesis of findings to develop recommendations for the next stage of the development of primary care networks.ResultsPrimary care networks have been implemented in a timely manner and have established a range of new local health services. Previous general practice collaborations provide much-needed support in terms of management, leadership and infrastructure, although they can be a source of tension within networks where interests, goals and ways of working do not align. Reasons for collaborative working typically focus on the sustainability of primary care and a desire for better-integrated services, although those cited as the basis for joining primary care networks were mostly related to policy and financial incentives. Early evidence reveals operational success in establishing organisational structures, recruiting to new roles and providing services as required by the national specification. Effective management and leadership, particularly with respect to having a committed clinical director, and constructive relationships between primary care networks and clinical commissioning groups, are important in ensuring success. In rural areas there was some perceived lack of fit with aspects of the primary care network specification, alongside existing challenges of providing primary care to rural populations.LimitationsArranging and carrying out interviews proved difficult given the workload associated with implementing networks and wider time pressures. Following the onset of the COVID-19 pandemic, the team was unable to undertake planned face-to-face workshops to explore findings. Given that primary care networks were in their first year of operation at the time of this evaluation, we were cautious in drawing definitive conclusions.ConclusionsKey lessons focus on increasing the engagement of general practices and wider primary care teams with networks;building leadership and management capacity to support networks in fulfilling their contractual obligations and meeting local health needs;and clarifying how primary care networks will operate in the post-COVID-19 health and social care system.Future workFuture work could involve evaluating the impact and effectiveness of primary care networks using quantitative and qualitative measures;undertaking research in both rural and urban areas, exploring the extent to which this context is significant;examining the cost and effectiveness of sustaining and extending leadership and management support within primary care networks;and understanding the relationships between primary care networks and the wider health and care system.Study registrationThis study is registered as PROSPERO CRD42 18110790.FundingThis project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care and Delivery Research programme and will be published in full in Health and Social Care Delivery Research;Vol. 10, No. 27. See the NIHR Journals Library website for further project information.

The impact of post-hospital remote monitoring of COVID-19 patients using pulse oximetry: A national observational study using hospital activity data.

Background: There was a national roll out of 'COVID Virtual Wards' (CVW) during England's second COVID-19 wave (Autumn 2020 - Spring 2021). These services used remote pulse oximetry monitoring for COVID-19 patients following discharge from hospital. A key aim was to enable rapid detection of patient deterioration. It was anticipated that the services would support early discharge, reducing pressure on beds. This study is an evaluation of the impact of the CVW services on hospital activity. Methods: Using retrospective patient-level hospital admissions data, we built multivariate models to analyze the relationship between the implementation of CVW services and hospital activity outcomes: length of COVID-19 related stays and subsequent COVID-19 readmissions within 28 days. We used data from more than 98% of recorded COVID-19 hospital stays in England, where the patient was discharged alive between mid-August 2020 and late February 2021. Findings: We found a longer length of stay for COVID-19 patients discharged from hospitals where a CVW was available, when compared to patients discharged from hospitals where there was no CVW (adjusted IRR 1·05, 95% CI 1·01 to 1·09). We found no evidence of a relationship between the availability of CVW and subsequent rates of readmission for COVID-19 (adjusted OR 0.97, 95% CI 0.91 to 1·03). Interpretation: We found no evidence of early discharges or changes in readmissions associated with the roll out of COVID Virtual Wards across England. Our analysis made pragmatic use of national-scale hospital data, but it is possible that a lack of specific data (for example, on which patients were enrolled and on potentially important confounders) may have meant that true impacts, especially at a local level, were not ultimately discernible. It is important that future research is able to make use of better quality - preferably linked - data, from multiple sites. Funding: This is independent research funded by the National Institute for Health Research, Health Services & Delivery Research program (RSET Project no. 16/138/17; BRACE Project no. 16/138/31) and NHSE&I. NJF is an NIHR Senior Investigator.

Patients' experiences of, and engagement with, remote home monitoring services for COVID-19 patients: A rapid mixed-methods study.

INTRODUCTION: Remote home monitoring models were implemented during the COVID-19 pandemic to shorten hospital length of stay, reduce unnecessary hospital admission, readmission and infection and appropriately escalate care. Within these models, patients are asked to take and record readings and escalate care if advised. There is limited evidence on how patients and carers experience these services. This study aimed to evaluate patient experiences of, and engagement with, remote home monitoring models for COVID-19. METHODS: A rapid mixed-methods study was carried out in England (conducted from March to June 2021). We remotely conducted a cross-sectional survey and semi-structured interviews with patients and carers. Interview findings were summarized using rapid assessment procedures sheets and data were grouped into themes (using thematic analysis). Survey data were analysed using descriptive statistics. RESULTS: We received 1069 surveys (18% response rate) and conducted interviews with patients (n = 59) or their carers (n = 3). 'Care' relied on support from staff members and family/friends. Patients and carers reported positive experiences and felt that the service and human contact reassured them and was easy to engage with. Yet, some patients and carers identified problems with engagement (e.g., hesitancy to self-escalate care). Engagement was influenced by patient factors such as health and knowledge, support from family/friends and staff, availability and ease of use of informational and material resources (e.g., equipment) and service factors. CONCLUSION: Remote home monitoring models place responsibility on patients to self-manage symptoms in partnership with staff; yet, many patients required support and preferred human contact (especially for identifying problems). Caring burden and experiences of those living alone and barriers to engagement should be considered when designing and implementing remote home monitoring services. PATIENT OR PUBLIC CONTRIBUTION: The study team met with service users and public members of the evaluation teams throughout the project in a series of workshops. Workshops informed study design, data collection tools and data interpretation and were conducted to also discuss study dissemination. Public patient involvement (PPI) members helped to pilot patient surveys and interview guides with the research team. Some members of the public also piloted the patient survey. Members of the PPI group were given the opportunity to comment on the manuscript, and the manuscript was amended accordingly.

The impact of remote home monitoring of people with COVID-19 using pulse oximetry: A national population and observational study.

BACKGROUND: Remote home monitoring of people testing positive for COVID-19 using pulse oximetry was implemented across England during the Winter of 2020/21 to identify falling blood oxygen saturation levels at an early stage. This was hypothesised to enable earlier hospital admission, reduce the need for intensive care and improve survival. This study is an evaluation of the clinical effectiveness of the pre-hospital monitoring programme, COVID oximetry @home (CO@h). METHODS: The setting was all Clinical Commissioning Group (CCG) areas in England where there were complete data on the number of people enrolled onto the programme between 2nd November 2020 and 21st February 2021. We analysed relationships at a geographical area level between the extent to which people aged 65 or over were enrolled onto the programme and outcomes over the period between November 2020 to February 2021. FINDINGS: For every 10% increase in coverage of the programme, mortality was reduced by 2% (95% confidence interval:4% reduction to 1% increase), admissions increased by 3% (-1% to 7%), in-hospital mortality fell by 3% (-8% to 3%) and lengths of stay increased by 1·8% (-1·2% to 4·9%). None of these results are statistically significant, although the confidence interval indicates that any adverse effect on mortality would be small, but a mortality reduction of up to 4% may have resulted from the programme. INTERPRETATION: There are several possible explanations for our findings. One is that CO@h did not have the hypothesised impact. Another is that the low rates of enrolment and incomplete data in many areas reduced the chances of detecting any impact that may have existed. Also, CO@h has been implemented in many different ways across the country and these may have had varying levels of effect. FUNDING: This is independent research funded by the National Institute for Health Research, Health Services & Delivery Research programme (RSET Project no. 16/138/17; BRACE Project no. 16/138/31) and NHSEI. NJF is an NIHR Senior Investigator.

Vertical integration of primary care practices with acute hospitals in England and Wales: why, how and so what? Findings from a qualitative, rapid evaluation.

OBJECTIVES: To understand the rationale, implementation and early impact of vertical integration between primary care medical practices and the organisations running acute hospitals in the National Health Service in England and Wales. DESIGN AND SETTING: A qualitative, cross-comparative case study evaluation at two sites in England and one in Wales, consisting of interviews with stakeholders at the sites, alongside observations of strategic meetings and analysis of key documents. RESULTS: We interviewed 52 stakeholders across the three sites in the second half of 2019 and observed four meetings from late 2019 to early 2020 (further observation was prevented by the onset of the COVID-19 pandemic). The single most important driver of vertical integration was found to be to maintain primary care local to where patients live and thereby manage demand pressure on acute hospital services, especially emergency care. The opportunities created by maintaining local primary care providers-to develop patient services in primary care settings and better integrate them with secondary care-were exploited to differing degrees across the sites. There were notable differences between sites in operational and management arrangements, and in organisational and clinical integration. Closer organisational integration was attributed to previous good relationships between primary and secondary care locally, and to historical planning and preparation towards integrated working across the local health economy. The net impact of vertical integration on health system costs is argued by local stakeholders to be beneficial. CONCLUSIONS: Vertical integration is a valuable option when primary care practices are at risk of closing, and may be a route to better integration of patient care. But it is not the only route and vertical integration is not attractive to all primary care physicians. A future evaluation of vertical integration is intended; of patients' experience and of the impact on secondary care service utilisation.

Early development of primary care networks in the NHS in England: a qualitative mixed-methods evaluation

ObjectivesPrimary care networks (PCNs) were introduced in the National Health Service (NHS) in England in 2019 to improve integrated care for patients and help address financial and workforce sustainability issues in general practice. The purpose of this study was to collect early evidence on their implementation and development, including motivations to participate and what enables or inhibits progress. This paper considers the core characteristics of PCNs, and how this informs their management.DesignA qualitative mixed-methods rapid evaluation was conducted across four case study sites in England, informed by a literature review and stakeholder workshop. Data collection comprised interviews, non-participant observation of meetings, an online survey and documentary review.ResultsGeneral practitioners (GPs) are motivated to participate in PCNs for their potential to improve patient care, enable better coordinated services and enhance financial and workforce sustainability within primary care. However, PCNs also have an almost mandatory feel, based on the national policy context and significant financial incentives associated with joining them. PCNs offer potential to bring GPs together to work towards common goals, deliver national priorities and respond rapidly to local needs.ConclusionsPCNs face similar challenges to other meso-level primary care organisations internationally, as they respond to local and national priorities and operate in a context of multiple goals and interests. In managing these organisations, it is important to find a balance between local and national autonomy, decision making and control.

Realist evaluation of the implementation and impact of the NHS carbon reduction strategy in the UK.

OBJECTIVES: To evaluate the extent to which organisational factors facilitate or inhibit the implementation of the National Health Service (NHS) carbon reduction strategy within acute hospital settings. SETTING: A single acute NHS Trust with four satellite sites which serve more than 2 million patients annually in Central England. PARTICIPANTS: Interviews with a purposive sample of 10 stakeholders, including those who conceptualised the intervention and those who were responsible for its implementation. INTERVENTION: The NHS is a major carbon emitter and therefore developed the 'NHS carbon reduction strategy (NHSCRS)' in 2009. NHS organisations are contractually obliged to develop a local carbon reduction strategy known as a Sustainable Development Management Plan (SDMP) which details carbon reduction measures (CRM), as described in the NHSCRS. However, the organisational context within which the SDMP is implemented is likely to determine the extent of its success. We undertook an adapted realist evaluation cycle to develop refined initial programme theories. Documents were analysed using thematic content analysis. Interview data were analysed using thematic analysis. RESULTS: CRM were most likely to be implemented if the Trust Board were sufficiently pressured by staff and reputational fears, and the potential impacts of CRM were perceived to align with wider organisational aims. Differences in implementation of CRM across hospital sites were related to logistical factors, accessibility to regional partners and contractual relationships. There were expected carbon, energy and long-term financial savings, with variability in the effectiveness of some CRM post implementation. CONCLUSIONS: Organisational factors, particularly Board leadership and internal implementation pathways, have a significant bearing on whether CRM are implemented or not. However, greater national support and guidance is needed for NHS organisations to effectively reduce their carbon emissions. Further cycles of this evaluation are necessary in multiple case study sites to illuminate the path to a net-zero NHS carbon footprint by 2045.

Remote home monitoring (virtual wards) for confirmed or suspected COVID-19 patients: a rapid systematic review.

BACKGROUND: the aim of this review was to analyze the implementation and impact of remote home monitoring models (virtual wards) for confirmed or suspected COVID-19 patients, identifying their main components, processes of implementation, target patient populations, impact on outcomes, costs and lessons learnt. METHODS: we carried out a rapid systematic review on models led by primary and secondary care across seven countries (US, Australia, Canada, The Netherlands, Ireland, China, UK). The main outcomes included in the review were: impact of remote home monitoring on virtual length of stay, escalation, emergency department attendance/reattendance, admission/readmission and mortality. The search was updated on February 2021. We used the PRISMA statement and the review was registered on PROSPERO (CRD: 42020202888). FINDINGS: the review included 27 articles. The aim of the models was to maintain patients safe in the appropriate setting. Most models were led by secondary care and confirmation of COVID-19 was not required (in most cases). Monitoring was carried via online platforms, paper-based systems with telephone calls or (less frequently) through wearable sensors. Models based on phone calls were considered more inclusive. Patient/career training was identified as a determining factor of success. We could not reach substantive conclusions regarding patient safety and the identification of early deterioration due to lack of standardized reporting and missing data. Economic analysis was not reported for most of the models and did not go beyond reporting resources used and the amount spent per patient monitored. INTERPRETATION: future research should focus on staff and patient experiences of care and inequalities in patients' access to care. Attention needs to be paid to the cost-effectiveness of the models and their sustainability, evaluation of their impact on patient outcomes by using comparators, and the use of risk-stratification tools.

The implementation of remote home monitoring models during the COVID-19 pandemic in England.

BACKGROUND: There is a paucity of evidence for the implementation of remote home monitoring for COVID-19 infection. The aims of this study were to identify the key characteristics of remote home monitoring models for COVID-19 infection, explore the experiences of staff implementing these models, understand the use of data for monitoring progress against outcomes, and document variability in staffing and resource allocation. METHODS: This was a multi-site mixed methods study conducted between July and August 2020 that combined qualitative and quantitative approaches to analyse the implementation and impact of remote home monitoring models developed during the first wave of the COVID-19 pandemic in England. The study combined interviews (n = 22) with staff delivering these models across eight sites in England with the collection and analysis of data on staffing models and resource allocation. FINDINGS: The models varied in relation to the healthcare settings and mechanisms used for patient triage, monitoring and escalation. Implementation was embedded in existing staff workloads and budgets. Good communication within clinical teams, culturally-appropriate information for patients/carers and the combination of multiple approaches for patient monitoring (app and paper-based) were considered facilitators in implementation. The mean cost per monitored patient varied from £400 to £553, depending on the model. INTERPRETATION: It is necessary to provide the means for evaluating the effectiveness of these models, for example, by establishing comparator data. Future research should also focus on the sustainability of the models and patient experience (considering the extent to which some of the models exacerbate existing inequalities in access to care).